<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <title>Resource change listeners in a concurrent world</title>
  <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
</head>
<body text="#000000" bgcolor="#FFFFFF">
 
<h1>Listeners and builders in a concurrent world</h1>
<font size="-1">[Draft copy] </font><br><br>
<font size="-1">Last modified: August 7, 2003</font> 

<p>In the context of the work on improving platform responsiveness,
there are some proposed changes to how resource change listeners
and incremental project builders operate.  This document describes and 
motivates these changes. The existing (Eclipse 2.1) behavior of listeners and
auto-build will be described, along with a description of how this will change in Eclipse 3.0.
Finally, this document includes a description of how existing implementations might 
be impacted and how they can adapt to the new behavior.
<h3>Overview and motivation</h3>
<p>
The principal proposed change for release 3.0 is that auto-build will be called in a 
separate thread from which the workspace change 
occurred.  This means builders will run concurrently with the thread that 
caused the change, and there will be <b>no guarantee at all</b> about precisely 
when builders will be called.  The only guarantee is that all builders
will eventually be run when auto-build is turned on.</p>
<p>
There are a number of motivations for this change.  First, there is a need to minimize 
the duration for which the workspace lock is held by an operation.  In Eclipse 2.1, 
every workspace operation had an automatic resource change notification and 
auto-build cycle attached to the end of it.  The operation did not complete until
all notifications and auto-builds had finished.  For the entire duration of this cycle, 
the workspace was locked to prevent concurrent changes in other threads.  
This long term exclusive locking must be reduced if there is any hope to obtain 
greater concurrency in Eclipse 3.0.  One way to shorten the duration of this locking
is to split up the operation itself and the auto-build into separate units of work.
</p>
<p>
In addition, this rigid notification and build strategy resulted in poor performance if clients
neglected to batch groups of changes into a single operation (via 
<code>IWorkspaceRunnable</code>).  Decoupling the workspace operation
from the build phase allows us to employ strategies to limit the number
of builds, and to lower the priority of builds to give precedence to other workspace 
changing operations.
</p>
<p>
<p>
Another change that will be made to improve responsiveness is to add notifications
<b>during</b> long running operations.  An upper bound will be introduced on the
delay between a workspace change and the notification.  This means that even if an 
operation takes several minutes, it will be reporting incremental resource changes 
periodically throughout the operation.  The increased notifications will act as an added
feedback mechanism to show the user that progress is being made on an operation.
This also allows the user to begin working with some of the output
of the operation before it completes.  For example, a user will be able to begin 
editing imported files before the import is complete, if the import is taking a long time.
</p>

<h3>Old behavior of listeners and auto-build</h3>
<p>Every workspace changing operation or group of changes batched with
<tt>IWorkspaceRunnable</tt> had the following structure prior to Eclipse 3.0:
<ol>
<li><tt>PRE_DELETE</tt> or <tt>PRE_CLOSE</tt> notification if applicable</li>
<li>Perform the operation</li>
<li><tt>PRE_AUTO_BUILD</tt> notification</li>
<li>If auto-build is on, perform incremental workspace build</li>
<li><tt>POST_AUTO_BUILD</tt> notification</li>
<li><tt>POST_CHANGE</tt> notification</li>
</ol>
The following characteristics are worth noting:
<ul>
<li>pre- and post- build notifications occurred regardless of whether auto-build was on.</li>
<li>pre- and post- build events contained a partial delta for the changes up to that
point.  Their deltas were always a subset of the delta supplied to <tt>POST_CHANGE</tt>
listeners.</li>
<li>pre- and post- build listeners are allowed to modify the workspace, but
other listeners cannot.</li>
</ul>
<h3>Overview of changes for listeners in Eclipse 3.0</h3>
<p>
With the proposed changes for Eclipse 3.0, steps 3-5 in the above structure are
removed from the operation.  The resulting picture looks like this:
<ol>
<li><tt>PRE_DELETE</tt> or <tt>PRE_CLOSE</tt> notification if applicable</li>
<li>Perform the operation</li>
<li><tt>POST_CHANGE</tt> notification</li>
</ol>
</p>
<p>
Periodically, the platform will perform a background workspace build operation.  The exact 
timing of when this build occurs will not be specified.  The structure of the build 
operation will look like this:
<ol>
<li><tt>PRE_AUTO_BUILD</tt> notification</li>
<li>If auto-build is on, perform incremental workspace build</li>
<li><tt>POST_AUTO_BUILD</tt> notification</li>
<li><tt>POST_CHANGE</tt> notification</li>
</ol>
The reference point for the deltas received by auto-build listeners will be different
from post-change listeners.  Build listeners receive notification of all changes
since the end of the last build operation.  Post-change listeners receive a delta
describing all changes since the last post-change notification.  This structure retains
three characteristics of resource change listeners that have been true since Eclipse 1.0:
<ul>
<li><tt>POST_CHANGE</tt> listeners receive notification of absolutely all resource
changes that occur during the time they are registered.  This includes changes made
by builders, and changes made by other listeners.</li>
<li><tt>PRE_AUTO_BUILD</tt> listeners receive notification of all resource
changes <b>except</b> changes made by builders and resource change listeners.
<li><tt>POST_AUTO_BUILD</tt> listeners receive notification of all resource
changes <b>except</b> changes made by other <tt>POST_AUTO_BUILD</tt>
listeners.
</ul>
However, there are some important differences with this approach. Prior
to Eclipse 3.0, auto-build listeners were always called before <tt>POST_CHANGE</tt>
listeners.  For this reason, the delta received by auto-build listeners was always 
a subset of the delta received by the <tt>POST_CHANGE</tt> listeners.
This relationship is now essentially reversed.  Auto-build listeners will receive
a delta that is a super-set of all deltas supplied to <tt>POST_CHANGE</tt>
listeners since the end of the last background build.



As before, auto-build listeners will be allowed to modify the workspace, and 
post-change listeners will not.
</p>
<h3>Overview of changes for builders in Eclipse 3.0</h3>
<p>
The proposed changes for incremental project builders are conservative.
The task of an incremental builder is to reconcile an arbitrary set of changes
to a densely interconnected web of source files to produce an output that 
is a consistent representation of the source.  This is a difficult enough task to achieve 
in a single-threaded environment that to attempt it in a concurrent environment 
would be courting disaster.  On the other hand, building can take a long time, and
there is a very high value in moving this work to a non-blocking background thread
to allow the user to continue working.
</p>
<p>
With these factors in mind, the proposal is to move auto-build into a background
thread, but to preempt the build when another operation attempts to modify the
workspace.  A typical work flow would go like this:
<ol>
<li>A user modifies a file and saves the change.</li>
<li>Auto-build starts.</li>
<li>The user modifies the file again and attempts to save.</li>
<li>The auto-build started in step 2 is halted at the nearest convenient stopping point.</li>
<li>The file modification is allowed to proceed.</li>
<li>Auto-build starts running again.</li>
</ol>
The &quot;nearest convenient stopping point&quot; in this case is at the end
of the current builder invocation.  No builder will be halted in the middle of building
a single project.  This strategy ensures that the user doesn't have to wait long before they
are allowed to make their change, and that workspace changes don't happen in the
middle of a single builder invocation.  When a preempted auto-build resumes, it must
start again at the beginning of the build order to ensure that interdependent projects
are built in the correct order.  However, builders that successfully ran
before will not be given the same delta again.
</p>
<p>
This new strategy is designed to have no impact on existing builder implementations.
In essence, this approach is equivalent to a mode where auto-build is off, but a
background thread is performing an incremental build periodically.  In other words, since this
strategy (apart from the preemption) can be implemented using existing API, it 
should have no impact on existing builders.  No new API is proposed for builders,
and no API changes are planned.
</p>

<h3>Summary of proposed API changes</h3>
<p>
From a binary compatibility view, this proposal includes no new API, and no
API changes.  The changes are 100% binary compatible with all prior Eclipse releases.
There are two proposed changes to the API contract of resource change listeners:
<ol>
<li>It will no longer be guaranteed that all resource changes that occur during the
dynamic scope of an <code>IWorkspaceRunnable</code> will be batched in a single
notification.  This mechanism can still be used for batching changes to avoid unneccessary
builds and notifications, but the platform may decide to perform a notification during the
operation if sufficient time has elapsed.  Note that it was previously possible for resource
change notifications to occur during the invocation of an <code>IWorkspaceRunnable</code>
via the <code>IWorkspace.checkpoint</code> API method.  Therefore, this API change
is not likely to be a breaking change for existing clients.  It is equivalent to the platform
deciding to call <code>IWorkspace.checkpoint</code> periodically during a long running
operations.</li>
<li>
The javadoc of <tt>IResourceChangeEvent</tt> states that <tt>POST_CHANGE</tt>
events happen immediately after <tt>POST_AUTO_BUILD</tt> events.  This will no
longer always be true.  The converse will still be true: <tt>POST_AUTO_BUILD</tt>
events will always be followed immediately by a <tt>POST_CHANGE</tt> event.
To boil it down: auto-build events still occur at the same time in relation to builds, they 
still receive exactly the same information, but they just occur less frequently.
</li>
</ol>
</p>
<h3>How clients might be impacted</h3>
<p>
From the viewpoint of a listener of <tt>POST_CHANGE</tt> notifications, this
new structure has <b>no effect at all</b>.  This new structure is
equivalent to a mode where auto-build is turned off, and the user is performing
a manual workspace build once in awhile.  Most importantly, <tt>POST_CHANGE</tt> 
listeners are still guaranteed to receive a delta that contains all changes made 
since the last notification.
</p>
<p>
For listeners to auto-build events, the new structure has some signficant differences.
Primarily, it will no longer be true that upon completion of a workspace changing
operation, that auto-build listeners will have been notified.  Client code that makes 
assumptions about when auto-build listeners run are likely to be broken by this change.  
For example, if an auto-build listener is updating a domain model to reflect changes
to the workspace, then this update might not have happened when the workspace
changing operation returns.
</p>
<p>
It is worth noting that only UI-level code can be affected in this way.  Core-level
code that is called via API may be called within the scope of an <tt>IWorkspaceRunnable</tt>,
so it can never be sure about when resource change listeners will be called.  The suggested
fix for this breakage is to use <tt>POST_CHANGE</tt> instead of build listeners if
it is necessary to have notification occur before the operation completes.
</p>
<p>
Another potential breakage is in clients that make assumptions about the ordering
of resource change listeners.  Previously, <tt>PRE_AUTO_BUILD</tt>, <tt>POST_AUTO_BUILD</tt>,
and <tt>POST_CHANGE</tt> events occurred in a strict linear order.  If listeners in 
<tt>POST_CHANGE</tt> assume that the auto-build events have already fired,
they may be broken.
</p>

<h3>Status and schedule</h3>
<p>
All described changes have been implemented. All work is currently in branch "Bug_36957" 
of project org.eclipse.core.resources. Basic sanity checking of the SDK reveals no 
significant problems, and some core team members are self-hosting with it.  
All platform core test suites now pass with background auto-building.  There are
some failures in JDT core test suites where they make assumptions about the timing
of auto-build deltas.  The JDT core team is investigating this.  The hope is to release
these changes to integration builds before the 3.0 M3 milestone.
<p>
Go <a href="../planning/3.0/plan_concurrency.html">back to concurrency home</a>.
</p>
(Plan item bug reference: <a
 href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=36957">36957</a>)<br>
</body>
</html>